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ABSTRACT: Accurate and early predictions of student performance could significantly affect 
interventions during teaching and assessment, which gradually could lead to improved learning 
outcomes. In our research, we seek to identify and formalize temporal parameters as predictors 
of performance ("temporal learning analytics" or TLA) and examine students' temporal 
behaviour during testing (i.e., in terms of time-spent). The goal is to specify a functional set of 
parameters that will be embedded in an adaptive assessment system in order to contribute 
towards the personalization of feedback services. In this paper, we present the motivation and 
rationale for our work, along with our methodology, initial results, contributions so far, and plans 
for future work. 
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1. INTRODUCTION 


Researchers attempt to identify and understand the most significant factors (i.e., with the highest 
impact or predictive capabilities) that affect student performance in order to use them in developing 
automated, data-driven, and adaptive assessment systems that support students to improve their 
performance. It is important to build systems that will allow students to monitor their own progress, 
support self-regulated learning, and help them evaluate and adjust their learning strategies to increase 
goal achievement. 

In our research, we have focused on the determination of predictive factors of student performance and 
on the discovery of causes of positive learning outcomes. More specifically, we explore whether 
associating students' self-perceptions (e.g., goal expectancy) with temporal trace data (e.g., total time to 
answer correctly) and coding their participation and engagement in terms of "time-spent" (e.g., un¬ 
certainty) could lead to a complementary dimension of a more concise predictive model. The research 
goal is to specify a functional set of parameters (latent, composite, or simple) that will be embedded in 
an adaptive assessment system that will provide personalized feedback services. 

2. LITERATURE REVIEW 

The most common indicators of performance explored by previous researchers include grades (in pre¬ 
requisite courses, during assessment quizzes, and in their final scores) and student participation, 
enrollment, and engagement in activity (e.g., login frequencies, number of chat/discussion messages, 
etc.) (Macfadyen & Dawson, 2010; Romero-Zaldivar, Pardo, Burgos, & Kloos, 2012). Researchers have 
also examined the effect of student response time on the prediction of their performance, modelled 
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engagement in terms of time that students are willing to spend during a course, and the relationships 
between study-time and motivation (Nonis & Hudson, 2006; Xiong, Pardos & Heffernan, 2011) with 
encouraging results. 


3. METHOD 


For the needs of our research, we configured a simplified version of the LAERS assessment environment 
(Papamitsiou, Terzis, & Economides, 2014). We implemented a testing mechanism and a tracker that 
logs student temporal activity data during testing. The collected dataset includes the following (for each 
student): student ID, the question the student works on, the answer the student submits, the 
correctness of the submitted answer, how many times the student views each question, how many 
times the student changes the answer, the timestamp the student starts viewing a question, the 
timestamp the student chooses to leave the question (saves an answer), the idle time the student 
spends viewing each question (not saving an answer, but choosing to see another question). We also 
embedded into the system a pre-test questionnaire in order to measure each student's goal expectancy 
(GE) (a measure of student goal orientation and perception of preparation; Terzis & Economides, 2011). 

Two case studies have been conducted so far: in the first, 96 Greek high school student participants (54 
females [56.3%] and 42 males [43.7%], aged 16 years old) attended a midterm computer-based testing 
exam. The 12 multiple choice questions of the test were related to the basic concepts of Informatics and 
Computer Programming. The testing procedure replaced the traditional paper midterm and was 
obligatory for all students. In the second case study, 259 Greek undergraduate university students (148 
females [56.9%] and 111 males [43.1%], aged 18-25 years old), participated in a computer-based testing 
and a computerized-adaptive testing procedure for the Computers II course (related to databases, 
information systems, and introduction to e-commerce). The participation in the midterm exams 
procedure was optional. In both studies, as external motivation to increase their effort, their score 
would count for 30% of their final grade. 

We adopted the partial-least-squares (PLS) analysis method (Chin, 1998) for the construction of a causal 
dependencies path diagram containing the structural and measurement model for the statistical testing 
of research hypotheses. 

4. RESULTS 


The following results are from the first case study only. The results for the research hypotheses are 
illustrated in Figure 1. The model explains the almost 63% variance (R 2 ) in Actual Performance (AP). In 
our model, Total Time to Answer Correct (TTAC) and Total Time to Answer Wrong (TTAW) are the 
temporal factors that have a statistically significant effect on AP. 
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Figure 1: Path coefficients of the research model, first case study 
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5. CONTRIBUTION AND WORK IN PROGRESS 


In our research, we configured the LAERS assessment environment in order to track student temporal 
activity data and compute the corresponding variables. We also embedded a questionnaire into the 
system in order to gather self-reported data. Initial results (Papamitsiou, Terzis, & Economides, 2014) 
highlighted a detected trend that TTAC and TTAW have a significant direct positive and negative effect 
on AP respectively, and they both have a significant direct positive effect on (un-)certainty. Moreover, 
GE is a determinant of TTAC and TTAW, and (un-)certainty is a determinant of AP as well. In a sense, (un¬ 
certainty seems to increase student effort to answer the questions. In addition, the detected indirect 
effect of GE on (un-)certainty indicates that student perception of preparation affects their cautiousness 
(Papamitsiou & Economides, 2014). 

Our goal was to explore the logged temporal trace data and identify their possible effects on student 
performance and on their predictive capabilities. To this point, the contribution of this paper is that the 
tools and methodology adopted so far have revealed the existence and the relationships of numerous 
temporal parameters. For example, Total Idle Time (TIT) is an indicator of (un-)certainty, while TTAC and 
TTAW imply the difficulty of the given task. As a next step, it would be interesting to apply process 
mining in order to investigate the nature of the temporal dimension of learning more deeply. We also 
aim to associate TLA to Weiner's attribution theory on causes of motivation and achievement (Weiner, 
1985). Further, we intend to examine emotional data collected during testing in order to discover 
possible relations to the existing temporal model for predicting actual performance. 

Currently, we collect temporal data (e.g., TTAC, TTAW, TIT, etc.), analyze them, and explore their 
capabilities in order to figure out whether such user-generated trace data might constitute an 
appropriate, cost-effective solution towards the personalization of computerized adaptive testing 
services. As a future direction, we are planning to embed temporal parameters into the LAERS system to 
guide recommendations regarding the next most appropriate testing item, and evaluate the 
performance and accuracy of this type of recommender system. 
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